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ABSTRACT 



Cognitive 'test scores in a large sample of American high school' stu- 
dents were analyzed in Public and Private' Schools . a report by James 
Coleman, Thomas Hoffer, and Sally KilgoVe. This latest "Coleman Report" 
came to strong conclusions atfcut 'the effects .of the public, Catholic, and' 
other-private schools: that the private sectors. produce higher test scores, 
and that they do so by employing better school Do'lic'ies'. 

We provide. a detailed critical evaluation of this material. We find 
that "the methods and interpretations employed fall below the minimum 
Standards of acceptability for social-scie*tif ic research! We also'find m 
that the strong conclusions are not warranted by the ev^d< 



ience < 
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THE CAUSAL ANALYSIS OF COGNITIVE OUTCOMES IN THE COLEMAN REPORT 



.1. INTRODUCTION •. * 

-■- ■ 4 — 4 

In their report to the^ National Center for Education Statistics [1), 

James Coleman, Thomas Hoffer, and Sally Kllgore (henceforth CHX) address 
( 

three questions with respect to cognitive outcomes (- test scores) In the 
American high school system (- 3 sectors: public, Catholic, and other 
private): (1) Do mean test secures differ across the thre^ sectors? (2) Do 
the sectors actually produce the differences in outcome? (3)"ftow do the 
sectors produce those differences — that is, by what policies do they 
accomplish their effects? CHK provide these answers: (1) Yes: mean test 
scores are higher; in the private sectors than in the public sector. (2) 
Yes: test scores are higher i^n the private sector even after controlling 
for differences in student characteristics. Furthermore, the private, sec- 
tors pr6duce larger gains in test scores during the last two years of\igh 
school than does the public sector. Furthermore, the Catholic schools come 
closer to v ttje "common school" ideal, by educating students of varying 
background more nearly aliVe tfth do the other-private and public schools. 
(3) The private schools produce their higher test scores by placing higher 
academic demands and imposing stricter discipline on their students than do 

the public schools. . , 

% t » 

Our task is to, assess the validity of these answers — that is, to eval- 

* ** 
uate the evidence ^nd reasoning that generated them. # have been handi- 

» * » 

capped by the style of the Report, a document of 233 pages + appendices, f 

Elaborate calculations from various regression equations are given, but 



the equations themselves are rarely presertted. Sample sizes *are* seldom 

indicated.* The definitions of variables^ are often cryptic. In such cir-^ 

cumstances one might want to rely on the .objectivity and scientific 

judgment of the authors as a substitute for the documentation one expects 

to find in a scientific 'report. But Aere is so much advocacy in the CHK 

• *> 

Report that this option was unavailable to us. We have, however, been 

- i . 

assisted by access *to some of ttie authors' computer output provided to us 
on request, and by access to unpublished studies by others who have been fc 
'reanalyzing the original data set. Finally' } we have tapped CHK^s^ar^tlcle, 
"Cognitive outcomes in public and private schools**" [2], and Coleman's own^^ 
restatement, "Private schools, public 'schools , and the public interest" 

* • c • 

[3]. ' , ' . 

Our summary assessments *are that the methods and f nterprerfat ions yised by 
CHK fall below the minimum standards for social-scientific research, and 



that CHK's answers to the questions posed\above are not warranted by their 



evidence. 



2. L MODELS AND DATA 
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A. -£ausal Model t * 

— " — — , _ j„ 

We begin wi F th a f ormul aX ion^ of the causal models implicft' in CHK's anal- 
ysis*. Figure 1A ^re^ehts a*general scheme fNo r school effects on cognitive 
outcomes whfch appears to underlie the text and tables in Chapter 6 of the 

* n • , 

Report. In our diagrams, x -'background, z - sector, s ^school 



policy -otesU score*, iHe straight arrows represent direct causal patVs; 
s # • 

wi th , the 'short ar^ws denpti,n# residual paths, and the curved arrows repre- 

\ x , ' t t i v 

sent nortcausal. associations . 5 The fi/st phase of the analysis 



Figure 1 ^ 
Causal Models for School Sector Effects ori Cognitive Achievement 




1A: General %chefne. 



7" 



IB: Test score by sector. 



J 




'ID: Senior test score by sector, con- 
trolling for sophomore test score. 




ICt Test score by sector.con- 
trollinp for background. 




IE: School policy by sector, con- 
trolling, for background. 




IP: Teat Score by sector, controlling for background and sch6ol policy. 

Synbols ; x s Oackrrrounjl, z J sector, s = school policy, y = test scor*. ' 
^otrairht arrows denote 'causal oaths, short arrov/s r.nnote -residual oaths' 
curved arro^/s denote noncausal associations. . 



\ 



"[1:154-161] takes up the^raw" association between sector and test 

score, asindioated in Figure IB. In the second phase of the analysis 

[1:165^180], the background-controlled relationship between sector and te$t 
f • 

score is investigated, as indicated. in Figure 1C. (Observe that as compared 

• * " * * 

with Figure 1A, the pa£h- f torn x to z has-been collapsed into an association 

^>etween x and z 9 a simplification which ig Justified when residuals leading 
to z and y in Figure 1A a^e uncorrelated; that is, jwhen unmeasured determi- 
nants of -sector choice are unrelated to unmeasured determinants of test 
score.) The second phase of the analysis [1:180-185] also contains an alter 
native attempt at controlling for differences between the students, in ✓ 

• % 

whict^ as effectively in Figure 10,.^^ (the change In test soore from 
sophomore to senior yealr) i's related to sector. The third phase [1:197-219] 
starts with an analysis of the relationship of school policy to background 
and sector (as in Figure 1$,) and proceeds to the -relation between test score 
and sector, now controlling for school policy, as well as background (as in 1 
Figure IF). 

w 

B. Sectors and Tes^ Scores 

Mean test scores by sector, al^ng with the standard deviations [1:154, 
TaMe 6.1.5; 2: Table 3] are reproduced here as Table U (Mote: fcere aad 
subsequently, we refer to both [l) and [2] as sources for the same tabular 
material.. In the event that entries differ between th^. two sources w we use 
those from [2]). There are three tests:. Reading, Vocabulary, and 
Mathematics (henceforth, R, V t M) contain respective^ 8, 8, and 18 

items. These test -scores are the pSiry dependent variables throughout 
CHK's study of cognitive outcomes in American high schools'. The tests, 



6 . 



^ Tabid 1 

Mean (and Standard Deviation) of Test Scores, 
by Grade and Sector A 



Public 



Catholic 



Reading 



) 



Vocabulary 



4 4 Soph Senior . Soph Senior 



~Z7 - 

3.60 4-. 48, 
(2.00) (2.10) 



4.34 '"^p.QQ 
(1.92j (l*96y 



3.69 4.48 

(1.88) (1.97) 

4.59 5.35 ^ 

fV-84) (1.74) 



Mathematics 
Soph Senior 



9.40 t 10.63 

<4.04) (4.24) 

11.05 12.10 . 

(3.56) (3.82) 



Other Private 



4.32 5.34 4.78 5.56 

(2.45 (2.04) (J. 00) (1.04) 



11*28 12.74 
(4.17) (4.14) 



Source : " [2: Table 3] . 



whiah are subtests of the longer tests that v*ere administered, are* clearly 
very short and are not generally used standardized achievement tests. What 
justifies their use as the measures of cognitive outcomes in the high * 
schools? ^.The Report is virtually silent on this question, except to say 
that - 

these tests do not cover subject matter that is an explicit 
part of the curriculum in the later years of high school. 



The mathematics items, are all rather elementary , i involving 
basic arithmetic operations, fractions-, and only a few hints 
of algebra and geometry [1:159}. - 



This passage shouldfcive pause to any reader who seek^s answers to questions 
(2) and (3). For it the mathematics description is any guide, the coverage 
of the tests is hardly an explicit part of the curriculum even in the 
earlier years of high school. They appear to measure the outcomes o*f 
elementary education' Thus, a face-value interpretation of Table 1 is that 



students who at tend private schools have had better^eleraentary sch6ol 

f 

achievement than^hose who atrtend public high schools. Perhaps the private 



elementary sector is responsible for this difference, perhaps not; that 
question is neither asked nor answered In the Report. Perhaps the private 
high school sector adds more to the elementary school achievement than 
does the publia high school sector, perhaps not; that question Is asked in 

the Report, but may not^be answerable with these test data. The R, V, M 

• t * 

tests are measures of the^inpurs into t^e high schools, not of value 

added; this has implications for the central issue of selection bias, an 

Issue which we discuss at length later. ^ 

There are three sectors, represented In the CHK sample hy 894 puhli'c 
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schools,. 84 Catholic schools, and $7 otjjer-prlvate schools. The very small 

dumber of other-private .schools' is disturbing, because that,'sector Is treated 

on a. par wl.th the other two throughout the Report. The extreme heteVo- 

genelty of the o thef-priva te "schools is remarked on by the authors tKem- 

• * . * ' * •' * 

selves: „ * 

> They Include the prestigious schools that' are often thought of 

s * . «• ■ ' 

, as the private -schools in America, schools that roughly .coincide 

' wlt b membership in-rhe National Association 6? Independent 

Schools. But they also include a wide range of church-related 

9 ' 9 *• I* schools, . . „ som$ of which operate on a shoe-string; and they 

x • % - include as well schools that have sprung up in response to .schoo.1 

desegregation policies and*p£her unpopular policies in'the public* 

, schools [1:1551. , ' 

Furthermore 1 , the randomness of this subsample is compromised by ' 

nonj^articipation. Although the Report is mute on this, we learn from' the - 

design manual [4*9] that of the 38 .other-private schools originally' drawn, 

only 23 agreed to participate, a number supplemented by an additional 

drawing of 4 cooperating schools. We doubt that this sample of 27 schools, 

nonrandoraly drawjT f rom an extremely heterogeneous sector, merits serious 

consideration. Population heterogeneity and small sample size would be 

reflecteU, of course, in large standard errors for v t'he sector statistics. 

Now consider the individual students vhose test scores enter Table 1. 

/The sample ^design called for obtaining 36" sophomores and 36 seniors 'in each 
' ' / ft/ • . . 

stfropl; but the actual sample sizes for the cells in Table 1 will be less 
* * * 

than th'ose target numbers suggest^. Small schools had less than the target 

number of students, some students wer,e absent, and others declined to par- 
tlclpate. By 'piecing together Information In [U] and CHK's computer output, 



• V 



are able to make* a Votygh estimate -of the numbers of students who 'actually 
took the tests. These an* ;displayed in <yjr Table" 2, along with 'the number 
of students. in the simply [1: AlGj • The substantial loss of observations 
(especially in the other-private sectoT), is furthejr cause for concern. 0 An 
obvious question is whether ..the %1 percent* of. other-private sophonfores vho 

did not take the tes^s would score as high as the 79 pefrcent who did. 

' ' f ' • 

Evidently, some of the raost^ elementary descriptive information^'about. the 

» * * 

tests,, the primary dependent variables of the entirt^cognitive-ouVcome 

stu^y, is missing; CHK, however, e*pres*s no reservation in their aaswer to 

question (1): Private-sector students have higher cbgn jUfl§v«r outcomes than 

do public-sector students • * In Table 1/ the differences, intest scores r 

across the sectors are s^nall in absolute terms, Vut large -relative to levels 

and standard deviations. CHK on occasion use the^ increase in average test 

score from the sophomore £o the senior £ear as a standard for describing Y 

' 4 : • . 

sector differences.* For example, on the- 8-item vocabula^ test, other- 
private sophomores answer one more item correctly than do. public sophomores* 
(4.78-3.,69 - 1.6-9), an increment jfhich -exceeds that«provided by the public 
. sctioo^s* in movinjg from - sophomore to senior year '(4 .48-;3,.60 - 0.88). 



Remarkably, CHK express no reservation about the validity of "nine-tenths- 
of-^a-'ques tioff" as an index of two Mars of educational attainment. 

9 



C. Statistical Inference 

* 

The crit ical parameters estimated by CfiK are regression coefficients for 
'the relationships represented by our Figures VC; ID, and IF* /There are 

• -'. < , * ' * 

surprising fcaps .in* their presentation of the estimates? most notably with 

• ' • ■ J . 

respect to> conventional * measures of reliability. Neijther standard errors. 



' * Table 2. 

* 

Number in Sample and Appra^xiraate Number 
and Proportion Taking Testis 



Public 

A 

Catholic 

-* 

Other Private, 



Number • 
in Sample 
Soph Senior 



T 



26,448 24,891 
2.R31 2,69? 
631 551 



• (2)' 

Number 
Taking Testff^. 
Soph Senior 



23,700 
2,700 
500 



(3) 

fc Proportion 

Taking Test 
Soph Senio^ 



21,500 
2,400 
450 



• 90 



.95 



,79 



r 



.86 



.89 



• 82 



Sources* Col. (1) from [UA10] { cols . (l) and (3) calculated by present 
- authors from [4r] and computer output provided by CHK. 



( 



nch: confidence intervals, nor tests of statistical significance are to be 
found in the CHK Report. - " 

The sample sizes show* in Table 2 are so large, howev**, that one might 
be tempfed to overlook the gaps on the grounds that everything roust be* # 
significant. Doing so % would be a serious mistake.- Ffcrst, In multiple 

regression analysis, collinea^ity among the explarfetory variables can pro- 

Jj * 
^uce large standard errors regardless of sample sire. Second, the High 

■< ' ' . 

School and Beyond (HS&B) sample was not a simple random 'sample. B? design, 

the students were not independently drawn across the high school .popula- 
tion, but rather In clusters by school. The effective sample size is 
consequently less than the number of .students, so that nominal standard 
errors computed on a siratfte-randora-sampl i'ng assumption must be adjusted 
upward. At the extreme,, if all students in each school were identical, the 
effective total* sample size wou^ld f*e the number of schools (1000 +) rather 
than The number of students (£8,000 +J. The Report [1 :A8 ] .indicates 
that nominal standard errors for univariate 4 statistics (e.g., means), should 
be multiplied by factors of 1.5 to 2.5. The appropriate ad justraent 'factors for 
regression coefficients have not be<?n determined. ^ 

'Third % the treatment of missing values merits attention. The HS&B s 

i 

sample contains many non-respons,es on Individual questionnaire items. For 
example, our reading of [4:8-97, R-?04J is that 15* of the students did not 

r 

report family Income, and 20% did' not report father's education. In a 
regression equation with many explanatory variables, the number of 
complete observations — those who have no, missing values at all — must 

4 f 

be-velJ below the nominal sample gize. Jay Noell [5], who ran test-score 
regressions with HS&B data using a long list of background variables simi- 
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lar to CHK's, reports that the number of complete observations^ the full 
sample is about 23,000, which is 40 percent of the original 58,-000. Douglas 
WiiLms [6] used a shorter list of background variables and was able to 
retain about 50 percent of the nearly 30,000 sophomores-: 

These considerations suggest that for CHK the task of obtaining 
appropriate measures of reliability wa*s ^lot trivial. The Report conta-ined 
'no information as to how pfssing values were handled; their article 
"[2:10] indicates ^hat the pairwise deletion method was used. ^The Report con- 
tained no standard errors at all; their article [2] now presents standard errors 
for various derived statistics. Those standard errors are computed on a random- 
sampling assuraption:^they take no account of clustering, and no account of 
the implicit reduction in sample size associated with missing values. Con-' 
sequently they understate true standard errors. As- a rough guide which may 
serve until proper measures of reliability are calculated and reported, we 

suggest that readers use a + 3-sigma (or + 4-sigma) rule-, rather than the 

t 

usual + 2-sigma rule, in assessing statistical significance. In any eveat, 
we have found no indication in the Report that the authors' interpretations 
and conclusions were. arrived at by applying conventional criteria of sta- 
tistical inference. 

J x ' 

As a f>f\al note on sample size, consider the number of blacks in the' 
other-private sector of the HS&B sample, a number unobtainable in the 

m t 

Report. Our reading of the computer output is tl^t the entire HS&B sample 
of 58,000 students contains lust 41 blacks in the other-private l sec tor . It 
is remarkable that OIK had no hesitation in calculating for the Report the , 
n^umher of blacks who would shift. to the other-private sector Ln response to 
a'unlversal $1,000 increase A n income [1:38], a calculation from which they 
infer, the response toSuition tax credits and school vouchers [1:68-73, 
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230-231*]. The inference is, ot^ fourse, , untenable; see Catterall and Levfn 
[7]. Nor has the*small sample'^ze for blacks inhibited CHK's publication 
of a racial segregation irjdtfx^or the other-private sector f 1-44; 2:Table ) 

2], an index which, one i\pw sees* measures the distribution of Just 41 

« 

persons across Just" 27 schools. 
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♦ 3. SECTOR EFFECTS ON TEST SCORES 

A. Background Sontro^ • 

There are various ways of presenting the differences in mean test scores 
across .sectors and across grade levels. CHK in effect proceed as follows. 
Let y fJ - mean test score i-n sector 1 at grade J, with i - 1,2,3 indexing* 
Jjie public, Catholic, and other-private sectors, and J « 1.2 indexing the 
sophomore and senior grade lj*yfels. Thenf define 

' %?2l * ?11 ' Increment (.at sophomore level) for' Catholic sector 

?3l " fll * Increment («at sophomore level) for other-private sector 

** " *>te * 

?12 * Til * Senior increment in-public sector 
A?22 ~ Y2l) • - Till - Extra senior increment for Catholic sect6r 

(Y32 - 73l) • (7l2 "-yn) ■ Extra senior, increment for other-private sector, 
* * » 

For each of three ^ests, these observed increments, or unadjusted mean dif- , 
ferences, as taken frow CHKHl: Table 6.1,3, 154; Table 6.2.'l, 171{ 2: 
•Tables 3, 4], are set out in the first column of our Table 3. 

r 

» 

- The issue now is the r extent to which these observed increments survive 
statistical control for initial differences among the students entering the 
several sectors and grades. As CHK put it* 
*t fT)he dif f ererfCes-^nay well be due merely to the differential - 

selection of different students into the different sectors, . 

i 



4'' . ■ « - . Table 3 

Yarlous Measures 6f Sector and Grande Effects on Test SCorls 



4) Extra, CathQilc *-.22 
..14 



5) Extra, Other- 
.prlva te 



VOCABULARY 

SoftKoaore Increment 

1) Catholic 

2) Other-private 

Senior Increment 

3) P^llc 

4) Extra, Catholic 

Extra, Other- 
private 

THEMATIC S > 



f 

So 



'Ophoraore Increment 

1) Catholic, 

2) Other-private 

Senior Increment 

3) Public . 

4) , Extra, Catholic 

5) Extra, Other- 
private 



.90 

i 

1.09 

> 

.79 

-.03 

>-.01 

1.65 
1.88^ 

1.23 
-.18 

.2* 



-.07 
.27 

.36 
.33 

.63 

"!/ 

.17 

.58 
.56 

.88 
.01 

.17 



-.13 
. .22 

.40 

.37 . 

.70 

•Q4; 

• 04 



.35 
.32 

1.02 
--02 

.1-4 



.41 

.31' 



.46 
.22 



*32 



Sources 


t 


Col. 


(l)t 


Col. 


(2): 


Col. 


(3); 


Col. 


(4): 


Col. 


(5): 


Col. 


(6): 



,41 
.20 

.21 



.38 
.30 

.60 



Rows 1, 2, 3 In [2:Table j 4];*Rovs 4, 5 are authors 1 
calculations from [2:Tablea 3,4]. * 
Rows 1, 2, 3 ih [2:Tahle>4]; Rows 4, 5 are auth/>ra f 
calculations from [2:Table 6]. 
Authora 1 calculations from [1:A12-A14]. 
From f 2 : 1 5] . 

, Authors 1 calculations (available on request V froa\6] , 
Col. (6): Authors^ calculations froo' l (2: Tabled]. 



* 






H : — 

Controlling 


For 






Observed 
(1) 


(2) 


Background 
(3) (4) 


Track 

*(5) 


, Dropout * 
(6) 


READIMQ 












•^ophomore* Increments 
1) Catholic \ 


.74 


.32 


m% .23 .26, 


.18 




* 

2) Other-private 


.72 ' 


.14 


- .02 






Senior Increment 
3) PuHllc^ 


.88 


.73 


- .75^ 




.47 
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' * • 1 
Are the differences entirely due to selection, or are there 

also different ef f ects. . .£[W]hat would be the differences ** % 

' . in outcome if the students coming into the different sectors ' % 

were alike? {1:167]. 

% 

s 

As their first statistical, control. CHK take a set of seventeen bapkgr^jjpd ' 

variables diNfen from the students* responses on the hour-long questionnaire* 

filled out in class [1:172; 2:9]. J 

A*, this stage, a. conventional /inalysis-of -covariande approach .would 

begin n#ith the regression, icross all students, of test score upon the . 

background variables and a set of dijmmy variables capturing the sector-by- 

, ■* , • 

grade classification. That is: 

' V • v A 

\ 3.2 ^ 

(1) .y - x'b + I w t w « tJ z^, 

where y - fitted test score, x - background vector, and* A 1 or 0 according 
to whether the student i-s or is ndt in sector i at grade j« (Throughout our 
'article, -vector" denotes a cpliwrnr^and the prime denotes its transpose; 




thus x'b should be read as the sum of products of the elements in x with the- 

• f * 

corresponding elements of b.) The slope vector b is taken to be the same 
for all sectors, while the intercepts represent the "main effects" of 
-sector and grade, ,The differences ( "cont/a-sts") among the a^j would serve 
as tentative estimates of ad justed- increment s to be compared with 




corresponding observed increments. If the focus were on sector effects at 



each grade Level, the next step in a conventional approach would separate 
out the grades and fit a pair of ^egressions, still running across all 
sectors: • . ) * 
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(2a) y - x f b 1 + t w a n z n (Sophomores) 

(2b) y - x'b ? + ^ a 12 « 12 (seniors) . 

I 

Here the slope vector Is *al lowed to differ by. grade level, while contrasts 

* * * 

among the a^j aga In estimate adjusted increments at each grade level^ 

Neither formulation (1) nor (2) is introduced in the Report/which works 

with the sample split into two sectors --^public and private (combining sec- 

tors 2 and 3) — and two grades. Four separate regressions (2 sectors x 2 

grades) are fitted, with dummy variables capturing the Catholic/other-private 

dichotomy within the- private sector; p i 

(3a) » y - x ' b 11 + \ Y . '(public sophomyes) 

?3b) y - x'b 21 + a 21 z 2i + a 3i z 3i <P rlvate sophomores) * . 

<(3c) y - x ' b 12 + a i 2 ' (public 'seniors), 

.(3d) y - x'b 22 + a 22 z 22 + a 32 z 32 (private seniors) . 

In this formulation, the impact of background on test scores is allowed to 
^differ between the publfc ^and private sectors (b]§ * b2 i ) , but not between 

r 

the Catholic aiyd other-private/ subsectors ( b 2j^ a ^3j)« 

These twelve' regressions (2 sectors x 2 grades x 3 tests)' are the only, 
ones tabulated in the CHK Report. The intejf^epts, slopes, and multiple R 2, s 

i 

are given along with the means of $he background ' variables in [1:A12-A14]. 

No standard errors are reported for the coefficients, so we cannot' as-sess 

% k 

the plausibility of. ttte point estimates of £he b's and a's. For example, is - 

high^famtly income "really" conducive to low test scores^ as 5* of ^he 12 
equations say? T)o other-private schools really depress teat scores relative 
to Catholic schools, as 4 of 6 equations say? 'Without standard errors, readers 



l'J 



"are unable to judge whether ^the reported sector differences in background 
effectfs (b2 j - b 1 j) t tffe teal, or merely attributable to chance variation. 
Indeed, with-no 1-nf orma tion on the fit of the simpler specifications (1) and 

' > ' 4b ' . 

;(2) above, there are no grounds for judging whether^the non-additive 
* i ^ v * 

specification^) provides a meaningfully better fit. Thij failure is com-, 
poundedby the authors 1 persistence in toeing ^11 point* est imates literally 
— that is, treating them as perfectly' reliable — except when sampling ' ' 
yaciability provides a convenient rat ionaliza't ion for anomalous results r ~ 
[1:41, 45, 46, 160, 177, 201, 203]. 

/< i ' . 

" Having fit equations (3), CHK face a mechanical problem in developing j 

estimates of adjusted increments. The sophdmore increments for the private 

sector, tor ^example, measure the vertical distance between the lines (3a) and 

(3b). Since the lines are not parallel, the an9we^ depends upon the value % 

of x at whic^i the distance is to be' measured . CH£ take the average t 1 * 

public school sophomore as the* reference 4 point . Formally, let x « mean ♦ 

» * ' ij 

value of vector x in sector, i at grade j, and p^j « predi/ted mean test 
score for sectoral at grade j. Pr6ra thei^ Equations (3), CHK calculate ' 



(4) 



j Pi 1 a ^T b ij + *ij. 



(with - b 2j) an< * process these predicted means Into adjusted increments 

• in the same manner that"th* observed means were processed Into observed 

^Increments (described at th.e beginning of this section). *pr example, the 

estimated adjusted increment $t the sophomore level for the Catholic sector 

*> * 

Js - p^. The results — which are^the content of [1:171, Table 6.2.1 
& 175, Table 6.2.2; 2: Tables 4 & 6] — are displayed In the^econd column 
( of our Table 1* ' ■ 
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CHK read these numbers as showing substantial rftf^erence-s to be attri- * % 
/ - \l 

buted to the private sector [1:173-174; 2:11] • In our^xperience' it is 

customary to cross-validate distance estimates from non-parallel regressions 

by using alternative reference points. CKK have not done so,* but 

the tabulations in* {1 : M2~iU) do permit us to' use tKeCverageWivate 

f sophomore (a mixture of x 2 i and x 31 ) as the 'reference point to obtain ?i y s 
Doing so, we obtain the third column of Tab}e 3, whic% gives a picture 
generally less favorable to the private sector. , S 

.Still less generally favorable are the estimates from fitting an addi- 
tive specification introduced for the first time in the CHK article, which 
shows the ai j estimates from equations (2). These directly give estimated 
increments at rile sophomore level, which we enter in the fourth column of 

• Table 3. The remainder of the column canftot be completed from their 
article because the model does al^w interaction by grade level*; but our 
/rough* calculation (details available, on request) shows that private, " 
sector, Increments a£ the senior level obtained from are uniformly ^and 
substantially less than those from CHK's Ck) • CHK [2:15] dismiss estimates 
of additive sector effects from ( 2 ) ■ as-Anf arlor , " on the grounds that one * 
must allow for a'ffill set of interactions between z and x. The conventional 
evidence for such a claim would be a tfeTfcnstra^tion that equations- (3) 'pro- 
vide a significantly better fit than equations (2), but as we have pre- 

viousjy noted, CHK offer no such evidence. * ^ 

l ' » 

Th6 alternative estimafes of background-controlled sector effects 

\ 

spread across the rows of columns 2, 3, and A of Table 3 span a moderately 

v 

wide range. As we shall see, .all of these estimates are upwardly 
biased. 

" 21 ■ ( • 



B.^ .Selectivity Bias 

CHK use 17 measured background variables to control for Initial dlf- 
ferences among the students Entering the several sectors. How adequate Is 

9 

this control We doubt that all 17 variables together can substitute for 
direct Initial \neasures of cognitive achievement, such as would be .provide^ 
by accurate R, V, and M test scores obtained just prior to entering high school 
Consequently, a mjjor- selectivity tjlas , problem appears, which may be concep- 
tualized as fqllows. Suppose that among students of the same measured back- 
ground, it Is the initially higher-scoring students who choose the private 
sector. Then the omission of those Initial* test scores, as In CHK's anal- 
ys^s^of covarlance, would produce. a selectivity bias In favor of the prlvat 
sector. The problem Is compounded here because CHK's outcome measures are 
themselves measuring pre-hlgh-school Achlevement^ather than the outcomes of 
high school experience. 

- CHK are wepl l*aware that In nonexperlmental situations*^ all ver— 
slons of "analysis of covarlance are subject 'to skepticism on the grounds 

that the covarlates raay'not capture all relevant preexisting differences. 

I 

In their words, 

[T]here may very well be other unmeasured factors In the 
self-selection Into the private sector that are associated 
? with higher achievement [1:224]. 

Hut (hey are oV^several K contradictory , minds^ about the efficacy of their 
measured background variables as controls for initial dl f f erefnce* . Thus CHK 
cite the known d i f f erence *ln motivation for education between parents who send 
their children to private* schools art* those who send their children to public 
schools, and flatly assert: 



19 . • 

.i - t- 

# * * 

. [T]h}s difference between parents, by its very nat\ure, is. 
not something on which students in public and private schools 
can^be equated. Thus ^his [background-regression] apprdach, is a 
•a particularly defective one in comparing public and private ' 
schools [1:168]. 

By the end of the Report, their verdict is altered: 

A large" number of background characteristics is introduced, 
to insure that the selectivity-related differences are 
controlled for [1:220]. „ 

How much insurance, , one wonders, can a particularly defective method buy? 
? oon a £ter the Report was issued, Coleman told an interviewer that by 
the use of background controls, 

bias resulting from self-selection was minimized. . . 

M If anythirrg, we probably overcompensated for the* " 

self-motivation lactor" ( New York Times , April 26, 1981: Y-19). • 

i 

t ' 

With underad justmenfc becoming correct adjustment becoming overad justment 
in three steps^, „it is evident that one cannot rely on CHK's assessments ' 
of their own method. 

• « 

Controlling far prior cognitive achievement* would- be the most natural 
approach to obtaining unbiased estimates o,f high school effects. An alter- 
native approach -would focus on the selection process Itself. By modeling, 
*nd eventually statistically controlling for, the syste'matlc determinants of 
seotor choice, one can estimate the net effect of sector upon outcome by 
reliance on ""tlne^cema ining sources of*variation in sector choice: see Barnow", 
Cain, and Goldberger [8J. The two approaches are related, and indeed 'in our 
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subsequent discussion we will not always distinguish between background 
variables as influences on prior achievement and as influences on sector 
choice. Omitting a background variable which is- y correlated* with initial 
achievement, but nc/t„wlth sector choice, should produce no bias In estl- 
mating sector effects. Similarly, omitting a background variable which is 

correlated with sjector choice, but not^ with prior achievement, sh'ould pro- 

■* 

duce no bias in Estimating sector effects. Thus full statistical control 

f 

over either initial achievement or sector choice would suffice. 

Consider now the specific set of seventeen variables which cpnstitute 
CHK's background vector. The list omits prior cognitive achievement, 
contains poorly measured background variables, and is far from comprehensive. 
Family income, for example, is obtained by asking the students to guess, 
in the middle of an hour-long questionnaire, the dollar bracket into which 
their family f s income fell. The student's sex and parents 1 occupations are 
on the questionnaire but are not included in CHK's lfst. 

The list iacludes several variables (e.g t , possession of an encyclopedia 
and of a pocket calculator) which are "not" clearly prior" to hi^h school 
achievement. CHK claim [1:170-171] that ' inclusion of such variables 
"overcorapensates" for pre-existing differences, and £hus tilts the balance in 
favor of , the public schools. But this claim is unfounded. High test scores 

might * lead to purchase of an encyclopedia, and the private sectrvr might pro- 

i — 

duce higher test; scores. But it is far-fetched to presume that this chain 

* 

of causation overrides the direct role of encyclopedia ownership as a proxy 
for unmeasured family background, and offsets other neglected prior dif- 
'ferances among^ttihents entering tfve several sectors. 

- Aibong the included background variables, only the two items referring to 
ea^hjparent's desires about the student's college plans directly measure 

24 ' 
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parental educational motivation. As for tfie student's ovm motivation,' 
nqthlng in the background list bears directly on academle vs. vocational 
• preferences. Private schools appear t ( o offer little In' the way of 

vocational training, as Is clear from data In the earlier chapters of * 
the Report^Jl :^80, 93, 97], / So vocational preferences, which presumably 
affect performance on cognitive achievement tejsts, may we 11 'influence sector 
choice as well. V We expect that among students of comparable background,' * 
those who are, oriented toward vocational and general curricula are more 
likely to be enrolled In ^he public sector, 

. Sector choice and test scores are also likely to be .influenced by such 
diverse and unrelated special disadvantages of students as mental or 
physical handicaps, or f oreign-lrfnguage-speaking hjoraes, none of which Is 
Included ln'the List, Indeed, none of the above-mentioned factors, apart 
from parental motivation, is discussed, l/t alone analyzed t In the 
Report In connection with the decision to enter a particular sector. There 
Is no discussion of the school administrators* admission and retention poli- 
cies We may presume that public schools ar£ the least restrictive but CHK 

. ^ v , . 

do not analyze or, discuss the private school^* criteria. 

The Report ^ totally silent on the several curricula — academic, 

general, and vocational — .taken by the stuSents, although this information 

♦ * ^ 

was on the questionnaire, and track is a variable that relates to both 
cognitive-achievement ancfsector choice. From Lutz ^rbring's report [9], 
•prepared at the National Opinion Research Center, and dated 'September' 1980, 
it was evident that in the HS&K sample/ the average academic-track public 
school student scored at about the same level. as the average private school 
• student. From computer output now available to us, we learn that the 

P 



distribution of student's by tract differs drastically across the' sectors. • 
According to the students' self-reports, the acaderoic/general/vocational mix 
is 20/46^22 in the publi v c sector, 61/32/6 in the Catholic sector, rand 
57/37/5 in the .other-private sector, at the sophomore level'. At the ^'enior 
level the corresponding f^ures are" 34/38/26, 69/21/9, 70/21/7, t Thul it is 
reasonable to. conjecture that had the student's track status been included 
as a- covariate — to capture initial abilities, proclivities, and interests 
— then the private-sector adjusted increments would have fallen substan- 
tially. 

.say, CHK's publications [1; 2; 3] are siLent on the track vari- 
able. Oueried just after issuance of the Report, Coleman 

rejected those who faulted him for comparing students in 
non-academic prograt*^ in public schools with those in 
private schools, where a higher proportion are in 
academic programs. "The program you. are in is not a 
'background' characteristic for which you should control 
. statistically," he said. "It has*a lot to do with school - 
policies" [ New York Times , April 26, 1981: Y-19] . 

we wouLd agree with the principle that track status is an inappropriate 
control variable when it is determined by sector policy, imposed as it wer* ' 
on otherwise, identical students, in-which case it should be viewed as one of 
the methods by whte*r the sectors produce cognitive achievement. By the same 
token, It is an appropriate control variable when 5 it is predetermined 
i/t the' sense "of reflecting initial student characteristics. A reasonable posi- 
tion is that it is a.Mxture of both. Coleman takes a polar stance, one which 
tilts the CHK -study toward overstating the private sector effect on test score's-. 
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In any assessments of Coleman's stance and the Report's neglect of track status, 
the following considerations are relevant. The variable in question is the 
student's own track status, as distinguished, say, from an index of whether 
or not pft school offered each.of the tracks, "fte latter variable might- be 
• construed a-s a pye policy, but not the* former. Further, Cbleraan's ex 
poat rationalization in effect views the general and vocational curricula as 
^ having no othef"f unction than cqgnitlve development'. Reader**vho believe 
that other functions are also served, such as preparation for life Careers, 
would have been helped by estimates of cofnitive outcome differences by 

track, as an index of the sacrifices made in pursuit of the other objec- * 

* * 

tlves. s — * 

In this connection, it is instructive to learn from Peng, Fetters, 
and Kplstad [10: ix] that the HS&8 

study's primary purpose is to observe the educational 
and occupational plans and activities of young people 
as they pass through the American educational system, 
and . takeuna their adult role's. 
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In any event, readers bf the CHK Report are. lll-served v by the omission of- 
all information on the empirical association between track and test score, 
information which; If present, they coultkhave interpreted in the light of 
their own judgments of exogenglty and endofeeneity. 

Some. Indication's of that empirical association in the* HS&B sample are now 
vailable. Millms [6] analyzed s.ophora^re^cor^es on the full reading and ' 
mathematics tests (18 and 38 items,, respectively).- Because of numerous 
mLssing values oh background variables /and the small sample sizfej In the 
private sector, Villms decided^ to use a short list of only f ive^ackground 
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variables to compare the, three tracks in the public and the Catholic sec- 
.torsi Furthermore/ because the students' self-Mpor^s of their track status 

• r 

ga^ve proportions which differed substantially from the aggregates'' reVorted 
by the^HS&B school administrators, he'decided to ^reclassify the" students. 
vFor this purpose, students who Reported themselves as on the general track,' 
but who also reported that they planned to attend a four-year collie *or 
university immediately following high school, were teassigned tathe acade- 

mic track. Thus WilLms's academic-track category is properly interpreted as 

t 

a college-bound category. This reassignment affected about one-quarter of 
the general-track students in the public sector, and about one-half of the 
generalitraf k students jtn the Catholic sector. > 

Villmd fitted separate regressions by track, with dummy variables intro- 
duced fdr sector. His sample sizes were approximately: academic track — 
6000 public, 1200 Catholic; general track — 4200 public, 200 Catholic; voca- 
tional track 2400 public, 100 tetholic. To summarize the results, we have 
c^rW-erted his estimates,, j*Mch are in long-test units, jM\to units for the 



^short tests, so that they are comparable to tho,s$ we have been discussing 
Toetatls available on request.) On that understanding, Willm^'s estimates 
of the Catholic sector increment 'at «the .sophomore level by track are: 



reading .12, .36, and* for the^ academic, general, and vocational 
tfetks, respectively; mathematics — .00, 1.13, anrf ^50 for the academic, 
general, y\d vocational tracks. Only the general-track increments are 
reported to be significant, and as Willms's notes, his standard errors are 
understatements because they neglect the clustered design. 

It is instructive to combine-*Wi*llms 1 s track-specific effects into'an 
overall estimate of the Catholic sector effect. For illustrative purposes 
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only, we do so by constructing a weighted average, using a mixture of 
public sector and Catholic sector weights. (Details of "our calculation are 
available on request.) We enter these In column 5 of our Table 3. Observe 
.the further ^shrinkage: the Catholic-sector Increments at the sophomore ' / 

level are, for reading, .18 compared with CHK's .32; and fof mathematics/ 
.32 compared with CHK's .58. 

Ofher evidence for the absence of a net sector effect on y> 
. test scores among students on' the same track Is available In a staff memo- 
randum prepareu at the National Center for Education Statistics [11]. 
Interestingly enough, the HS&B tests analyzed there are two which tap mater- 
ial that Is In fact taught In high schools, namely "Science" (for sophomores) 
and "Mathematics ir ^ fo ? sophomores and seniors). In the NCES memorandum 
mean scores for academic-track students are tabulated for cells defined by 
two sectors (public and Catholic), three ethnic groups (white, black, Hispanic^ 
and three socioeconomic status ( SES) 'oategor les . We have constructed 
a weighted average across ethnicity and SES for each sector. We find that among 
sophomores, the Catholic sector has a positive increment for mathematics^ 
and essentially a zero Increment for science; among seniors, the Catholic sector 
has a negative Increment for mathematics. (Details available on request.) r 

The Report does not use, or mention, the widely used econometric 
approachjB^selectivity bias in nonexperiraental data, the gist of which 

m 

is a multi-equation model in which the outcome equation is supplemented by 
an equation determining selection (i'e., sector choice). This model expli- 
citly captures the possibility that outoome and selection ma? have common 
measured and unmeasured determinants. Under restrictive conditions, estimation of 
the multi-equation model will provide unbiased estimates of the net sectfor 
effect. At a minimum, it will provide some guidance as-to'the extent to 
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ich test-score differences might be attributable to preexisting dif- 
ferences rather thaji to sector effects. In fact, the fir^t step of the econ- 
ometric approach, namely a probit regression of a student's sector status 
on measured background, would by itself provide an informative sifiusary 
of the measured background dif^fences among students entering the sec^rs. 
The approach Was developed several years ago by the economist James Heckman 
[12J, a colleague 'off Coleman, Hoffer^ and £ilgore % at NORC. 

Noell.[5J has applied several versions of this approach to the HS&B data, 
using the full reading and mathematics tests for both gVade levels. His 
first-step probity regressions indicate strong effects of background upon 
sector choice, with fegiop, religious background, and the. students' college 
.expectations as of «Bth grade — variables not included in CHK's background 
vector — being- among the most significant. His second step results — the 
adjusted regressions of test score on background — are mixed. The private 
sector effect becomes negative at the senior level, but becomes more posi- 
tive at the sjpphoinore leve^l. The magnitude of these estimated effects is 

sensitive to the speci f icafe ion of th« test-score equation, specifically to 
- * 

the .inclusion or 'exclusion of Catholic religious background as an explana- 

* ** * * . 

tory variable* 'Evidently, this f^rst application of the Heckman approach to 

i* t ' * 

the HS&B data set has not provided a definitive resolution of the 

V ' \ ■ ' 

selectivitj-bi^sftiasue. At this sta^e, it appears that "strong" conclusions 
from the HS&B data get ate not robust across plausible changes in model 
Specification. x • 

In empirical applications of the econometric approach, two persistent 
problems -are (1,) tire specification af "exclusions" for the outcome equation, 
and (ii) high cpliinearity when such exclusions are not imposed. (See 
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Barnow, Cain, a nd Go Id Merger [8J>. Here the exclusions refer to variables 
which affect sector choice, but do not\affect test score. If such variables 

t 

are picked out on an ad hoc basis, or capriciously, the estimates of sector 
effect will'not be. val4n\ On'the other hand, If no exclusions are imposed, 
the variable constructed in the probit-regression step, as a function of the 
measured background variables,, is likely to be highly collinear with those 
background-variables, when they aril reappear in the tes t-scor^^iation. -If 
so, estimates of sector effects will.be unreliable. Others who follow 
Voell's lead in using the econometric approach may find that^he HS&B data- 
set does not contain a rich enough set of measurements. If so, the 
selectivity-bias issue will remain unresolved. 

k 

C • Background Revisited: The- '"Common School" 

CHX* [1:176-180] degress from their^foain focus on the rafcan differences 
across the sectors to , invest iga te the interaction between sector and the 
students' backgrounds. Thfc? digression is worth some attention for it led 
the authors to a statement that was emphasized in press coverage: 

Altogether, *£he evidence is strong that the Catholic schools 
function much closer to the American idea of the "common 
school," educating children from different backgrounds alike, 
-than do the public schools ... Catholic schools -nore nearly 

i 

approximate the "common school'" ideal of American education 
than do public schools, in that the achievement levels of 
students from d i f ferent parental educational backgrounds, of 
black and white students, and of Hispanic and non-Hispanic J 
white students are more nearly alike, in Catholic schWs than 
in public schoels, fl:221, 232]. 
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What is the statistical basis for this statement? Regression of test 
score upon measured background by sector is again the mode of analysis, * 
despite the authors' v pr£vious dictum that this method Is "particularly 
defective." But now background is % confined to only 5 of the 17 variables* 
namely family income, father's education, mother's education, race (black 

vs. white), and ethnicity '(Hispanic vs. non-Hispanic), awhile all three 

• «* 

sectors are distinguished. 



For each test» 6 regression equations (3 sectors x 2 grades) are / 

f , ■ • 

fitted: 

(5) ^ y x'b^ V ai j (i - 1,2,3; J - 1,2) ' ' . S ^ 

/ 

where y • fitted test score, x - b>ck$ round vector (5 elements), the b^j are 
slopes, and the a^j are intercepts. CHK [1: Table 6.2.3, 178] present 
* selected elements of the b^j estimate namely the coefficients on the race 
dummy, the ethnicity dummy, and on^ combination of the parental education 
variables. (The coefficients on income are not given, nor are the race and 
jt e^hnicit> ^coefficients for the other-private sectors, the latter "because 
the numbers of blacks and Hispanics in the sample of these schools is small 
enough to make estimates unstable.")' At the sophomore level, the Catholic- 
sector coefficients b2i are smaller (closer to zero) than the corresponding 
public-sector coefficients b^. Thus, fjr example, 

The achievement of blacks is closer to thAt of. whites... 

in Catholic schools than {n publi? schools [1:178]. 
♦ -» 

Also, in the Catholic sector y the senior-level coefficients are generally 
smaller than the corresponding sophonvo re- level coefficients, while the 
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reverse is tme in "the p^ibHc^sector. Thus, ^ 

^ jN]ot o,nly is achievement roo^ alike among students from, 

• different backgrounds -in the Catholic schaols than in\ the 

other sectors, It becomes increasingly alike^frora tft)e * 

sophomore to the senior year f T : 179-1-80] . - 9 
• % 

> » * ^ 

This then is the empirical source "common school" conclusion* 

It is hard^to treat this material seriously. First, discarding 12 of 17- 

explanatory variables^ may be expected'^) distort the estimates of the, 

remaining 5 coefficients^ The potential for bias would be Indicated by a 

2 

comparison of R *s f-rom regreasions using the lan| and short lists of 

background variables. The , Report is silent, but CHK's computer output 

s ^ 2 

indicates that discarding the' 12 variables reduced the multiple R 's by 

* (• • % i ' 

abou t .one-third and substantially changed regression coefficients on the 

x \ • . 

retained variables. Second, reported differences between public-sector and 
Catholic-sector coefficients might be attributable to chance variation. 

A third line of argument suggest^ that the smaller slopes in the 
Catholic sector are an artifact of selection. A general result In regres- 
sion analysis is that selection on t^xe dependent variable tends to attenuate 

slopes. And our previous discussion has already Indicated that the Catholic 

s 

sector appears to be selective, in the statistical sense, of students -with 
favorable measured background and unmeasured academic/abilities. (A similar 
situation will arise If one considers the public school acade*mic track as 
analogous to a private school. One would expect to find flatter rela- 
tlonships of test score on background within academic track than across all 
tracks.) More pointedly, the selectivity argument suggests that students 
with unfavorable measured backgroun/s who nevertheless enter the private 
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sector are precisely those whose unmeasured academic proclivities were 
unusually favorable, which is a restatement of the flat-relationship resJult, 
(For a formal discussion of conditicrr^ under which selection guaran^^s * 
attenuation of slopes, see Goldberger [lJ].') * 

On balance, it is evident that CHK's "common school" conclusion has no 
solid empirical support. » 

I . 

B. Senior Increments . . 

To assess sector effects on senior test scores, CHK [1:167-169] intro- 
duce sophomore test scores as an alternative to background measures for the 
statistical control. More precisely, they suggest that sophomtfre-t o-senior 
change, - y^, syves as a measure of outcome produced by the schools. A 
plausible model at this point might be 



(6) 



y 2 ' 8 o"l + V 8 1 *V 6 2 + hll °12 Z 12 +,u - 



where and are the sophomore . and senior te^^#cores, x 1 and x 2/ are the 
sophomore and senior background vectors, u is a residual, and the^x^ are 
intercept* (whose differences would be read as -sector increments). In 
the light of (6), CHK's direct use of change scores as estimates of the 

a requires the stringent assumptions that 8 - 1 and 8, - 8„ =» 0. CHK Fl- 

At o 1 1 2 9 1 4 

169] remark on the first of these,' and suggest {2-12] that ruling out an 
interaction between level and change (that is, Imposing 8 - \) tends to bias 
t'heir results in favor of the* private sector. (The rationale apparently Is 
that 8 o > 1, and y 1 is higher in the private sector. Why 8 q should be 
greater than unity is not clear to us.) On the other hand, they do neglect 
background entirely In tjpls phase, suggesting [2:12] that the control Is 
not necessaryT*-It seems to us that omfttlng x T and % 2 does bias the S 
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results in favor of the private sector. For example, a family breakup pro- 
ducing a single-parent household might occur mpre frequently among seaiors 
in the public sector and concurrently reduce achievement.* If so,, by ignorir 
the initial and subsequent levels of x, CHK' s* approach would tend to pena- 
lize the public sector for the decline in y^* 

As it happens, equation^6) cannot be fitted at the individual student 
» * — * 

level in the HS&B sample. Neither y Y nor x Y is available for individual 
seniors, because the sophomores and seniors are not the same* persons. 
(This limitation will be removed when the second wave of the HS&Bsurvey 
returns to the sample schools two years bfter the first wave.) Equation 
(6) could have been fitted at the level of school averages, but CHK 
proceed directly to the sophomore-to-senior gains, averaged by sector 
[1U$0-185; 2:11). 

The observed* average gains are in fact the observed mean 
differences y n - y ^ (i .« 1,2,3) which were previously processed into the 
increments given in our Table 3 (lines 3, 4, 5 of the first column). Notice 
that the "extra senior increment" for the Catholic sector, relative to the 
publjc sector, is negative for all three tests. At this point, CHK call 
attention to a particular selectivity bias problem with the y ^ The senior 
test score distributions are distorted by the presence, t»r rather the 
absence, of dropouts. From the rosters of the sampled schools, they esti- ' 
mate that as a consequence of students having dropped out of school during 
Ithe last two years of high school^ 317 of the senior class is missing in the 
public sector, 13% in the Catholic sector, and 157 in #he other-private, pec- 
tor. (These rates are tod high. Official NCKS statistics [14:15] indicate 
that the national dr6potit rate is 20%, which is we.ll below the national 
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f igur^implied by CHK's Vector jrates, namely 29Z.) Presumably it is the 
lover-achlevlngs_tu^rrtT who tends to drop out, so the observe^ ov^state 
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the means for the full senior class, the overstatement being largest in the 
public sector where the dropout rate is highest. ^ 

CHK's solution to this serious problem is to produce for each sector a 
mean for the hypothetical full senior class, y** and calculate drbpout- 
adjusted sophomore-to-senior changes, y* 2 ~ y . These are given in the 
upper panels^of CHK's Table 6.2.5 [1:184] a^nd Table. 5 [2)-.' Processing them 
into increments gives the sixth column of <5tir Table 3. 

The public sector effect ("senior increment") is reduced substantially 
as compared with the observed figures in column 1, and even as compared with 
the background-ad jus fed figures in column 2. The Catholic sector effects 
("extra senior increments/.) are now non-negatWe, and the extra senior 

A 

increments for the other-private sector have become more positive. It is 

just these figures which led CHK to 

the conclusion that greater cognitive growth oceurs between 
the sophomore and senior years in both private sectors than 
in the public sector [1:225], 



reversing their earlier view that the observed growth seemed "very small 
everywhere" a^fd "very much the same among the different sectors" [1:158]., 
Upon examination, " It was faulty me thodology. that generated their find- 
ing. CHK describe their calculation of the hypothetical y*^ as being 
% 

based on the assumption that 

the dropouts came from the lower 50 percent of the test score 
# % distribution on each test and were distributed in that lower 
half In the same way that remairiiftg seniors In the lower, half 



9 

ERIC 



33 

♦ 

are distributed. What this means in effect is that within the 

lower half of the* senior test score distribution, and within 

the upper half, the distributions do not change; 'but the lower 
i • / 

•half ,-augmented by the dropouts, becomes a lateer share of the 
» T 

t otal • J • 

This assumption probably errs on the side of being 
' favorable* to those schools with hi^i proportions of dropouts 
(in this case public schools), because dropouts are probably 
concentrated more'toward the bottom of the distribution v than 
is assumed. Thus the assumption is probably conservative 
witJ respect tp the inference at hand: that is, the greater 
achievement growth in the private sector f 1 : 182-183] . 

We confess to some uncertainty as to'the arithmetic operation being 
described. Our best guess is that yj^s the score which 100 t{% of the 
observed seniors exceed in sector i, where 

(7) t --l , - > 

1 2d-w t ) 

with w t belng*the dropout rate for sector 1. A rationale for this estlma- 

* 

tor would run as follows. Suppose that the full population test score^ 
distribution were -symmetric, so that half were above the mean. If the pro- 
portion w t o5 that population drop out, all from trhe lower half, then the 
upper half of the original* group would constitute the proportion t t of the 
, surviving seniors. With w T - .31, w 2 - .13, w 3 - .15, the upper half of the 

- — r : original groups woul d. const t tute ti - .72, t2 - .57, and t3 » .59 of the 

surviving seniors in the public, Catholic, and pther-pr ivate sectors respec- 
^ ' tively. Thus yf 2 is taken to be the score such that 72% of the observed 

ERIC ■ , 3- 
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public seniors score above it, etc. 

If this is CHK's estimator, then -they did not at all err on the side 
of being favorable to/£he public sector. For the validity of t.he construc- 
tion above as an estimator of the mean rests only on the assumption that 
^ropouts came entirely from the lower half of a symmetric distribution, and 
not on whether they were uniformly distributed Across that lower half or 
more concentrated toward the lower tail. < f 

More important, it is evident that CHK are ad Justing , the wrong distri- 
butions in their dropout procedure. To the extent that dropping, out is 
determined by background, it is the conditional distribution of senior test 
score given background, rather than the unconditional distribution, that 
demands adjustment. Recall that the background controls have been dispensed 
with in thii portion of the CRK analysis. 4 Surely, the measured background 
factors — family income, pajrental education, both-paVents-present , 'et^h- 
nicity, and the parents' thoughts on whether the student should go to college 
— are 1 predictive of dropping out, as well as of senior achievement. CHK 
have, in their change measure, reintroduced the very background differences 
they had been so insistent on controlling for in the regression phase. 
^ A formal specification and estimation of the dropout process might be 
based on the econometric approach, mentioned earlier. Without it we have 
no definitive ^gures for the sector eft^vff if. any,, which would remain - 
after controlling both for background and for* dropping out. But let us : 
recalj^hat CHK's initial 17-variable background regressions already pro- 
vide<J adjusted measures of sophomore-to-senior growth in the several sectors, 
given in rows 3, A, 5. of -the second cokimn of our Table 3. To the extent ' 
tbat dropping out does depend on background, those figures can now be viewed 
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as controlling both for background's direct effect on outcomes and for its 
effect on* dropping out, I^tfdeTd, CHK coufd agree with this notion, for they \ 



wrote: 



[FJamily backgrounds of seniors are slightly higher than - 

th^fsV of sophomoYes, a difference that is attributable to 

greater dropout rates ... for students from lower backgrounds [1:1/3]. 

According to the second column of Table 3, the public sectojr has larger 
growth than the Catholic sector for two of the three tests, while growth in th'e 
'other-private sector is moderately greaterthan in the public and Catholic, 
sectors. In the Report, CHK [1:175] caution against using those figures 
because of their negl*ect of the dropout phenomenon. The figures they pre- 
ferred were the dropout-adjusted ones, which we now see are^biased'. In 
their article, CHK appear less certain: the bac>gr6und-ad justed extra 
senior Increments are now introduced after, and as a "variant" on, the' 
dropout-adjustment method of assessing-dif ferWial gain*,, and are said to 
suggest that the "dropout correction may have betfn too great" [2:14-15]. 
Nevertheless, a paragraphs so later their reservations , are abandoned and 
frhey return to the strong language of the Report: "considerably greater" 
and "sub^ntial" differences in favor of the private sectors. 

The background-controlled increments are arguably preferable to the drop- 
out-adjusted increments, but selectivity bias remains ^as a confounding fac- 
tor. As CHK remark, since 

dropping out of school is an act of negative selection, 
N> - the students who 'drop out are very likely lower achieving' 

than those from similar backgrounds' who reraafn in school* 
, [1*1811. ' ■ " 
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The bacJcgroun-d-controUed increments would seem to be more upwardly biasp^ 

in the public sector if it were to turn out that this sector has a higher 

dropout probability for students of comparable measured background. (As 

matters now St and, there i* no information on this issue in the CHK reports.) 1 

However, if students were fully comparable in their backgrounds, allowing 

for the upmeasured^as well as the measured determinants, it is not at all 

obvious that the dropout probaWJUty for the public sector would he higher. 

As a generalization, parents who anticipate that their children will drop 

* • 
out are unlikely to pay the extra expenses required for private) schooling, * 

t so private-school ^nrollees have a retention propensity th^kref lects 

parental motivations that are "like those previously discussed in connection 

• f 

with test-taking abilities. It is conceivable that, after controlling for t 

this retention propensity as well as for the available background 

/ 

variables, the dropout rate In the public sector would belover, precisely 

because public school policie's may be more lax, thus encouraging retention. 

§ 

In this light, using the background-controlled sophomore-to-senior changes 
may not be biased in favor of the public sector. 

CHK carry out further calculation© to convert the senior increments 
(-additional number of test items answered correctly) into ^annualized gpowth 
rates. ' To the two points y* 2> y^ for each sector they fit a differential 
equation, the parameter of which, interpreted as the ^te of growth g, is 
estimated as 

(8) . log (1+g) » (1/2) log ((T-^ 11 )ftT-y} 2 )J 

where T - total number of items on th^ test maxioKim possible score). 
'These growth rates are the entries in the lower p/nels of CHK's Table 6.2.5 
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Various Measures at Growth Rates in Test Scores, 
SqpHfonore-torSenior, by Sector 




Dropout Adjusted • Background Controlled 

Cflling Formula c|fcvenMonal* Ceiling Conventional 
+ formula Formula Formula 



v \ Log Arithmetic v ' Log 



(0 



(2) 



(3) 



Log 

(4) v> 



Log 

. (5) 



READING 

Public 

C^holic 

Other-private 
VOCABULARY 
r Public. ' * 

CathoH<^ 

Other-.private 



.06 


.05 




. .06 


.09 


.10 


.07 . 


.06 

* 




\ .OS* 


»\'09 


- .08 


.13 * 






, .09. 


r .14 


.11 




.05 


.04 


• 


'.05 


.10 


.09 




.06 


.11 


.10 


< t 


-ol 


.02^- 




f 


.02 


.05 


.05 




.03 



.08 *>» .07 



^04 



.08 

--r x 

.12 
.12 

.06 
.06 
.07 



.08 
.10 
.09 

fi05 
.OA 
.05 



Cols. (1): > Calculated by applying our equation (8) to tipper panel of 
Table 6.2.5 Jl : 184] . In principle, figures should coincide 
^ with those in the lower panel af that table., The slight A 
discrepancies are apparently attr ibutable 'to rounding. - '\ 
CMs. (2) and (3);' Calculated by applying our equations M) and (l^>> 
respectively to upper panel of CHK's Table 6.2.5 [1: 184] . " * * *a 
Cols. (4), (5): Calculated by applying f ormulas^correaponding to ojur 
Equations (8) and (10) tespectively to data in CHIC f2:?ableg 4^6>.. 
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V 

t 1 ; 1 84 ] and Table 5 [21, reproduced here as the first column In our Table 4. * 
In their fornuj^ion, the growth rate is in* effect ttie percentage decrease 
•in the number of incorrect answers, ^ssentiallx the' same figures emerge 
from the following arithmetic variant of CHK's ^rmula: * * 

(9) g - (i/2)(y* 2 - y^/a-y^); 

9 

V 

see the second column of* our Table A. k *ore conventional growth rate would be 



>nal 
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defined in effect/as the percentage increase in the numbl^of correct answers. 
Calculating this as ^ § 

(10) , log t fl+g)'- (172) logCyJj/y^)^ * 

gives tjie numbers in the third column of Table A. The fourth and fifth * , 
columns of Table A come .from applying (8) and (10) ^respectively to the back- 
ground-adjusted, rather than dropout-adjusted, increments. 

ft 

Evidently, the choice of growth rate measure is a ftontrivial matter. ™ 
Columns (3) and (5) in Table A are less favorabll* to the private sectors 
than their counterparts (1) and (A)v CHK's rationale for loosing their 
distinctive growth rate formula is that it corrects fjor ceiling effects 
(high-scoring students are limited irfthe number of additional correct 
answers they can get). Actually, their cho-ice is an arbitrary one from 
among the alternative devices for correcting *a deficiency of the test- 
instruments they used, and one whTTh, in the- event, tilted the balance 
toWar^i the private sector. 

To summarize our assessment of the CHK analysis of senior increments: 

i * * ■ 

find no evidence of ? jposiMve private-sector effect on growth front 
r 

sophombre' to senior year. Contrary t6 CHK's post t ic>fty s>> t*eir "extra senior 

f 

^ ' ' • * ' 42 V . ( 
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lnc ] re < >ents " are * contaminated by selection bias, their dropout adjustment is" 7 
inappropriate, and their growth rate foTHnula is arbitrary. In each in- 
stancy, their choice has tilted the balance in favor of the private sector. 

Our summary stands in sharp 'contrast to that by CHK: 

* * %» \^ 

The estimated learning rates show great differences between 

; students in other privatji schools and those in public * 

schools .... It is true that various assumptions are necessary, 

^ as discussed earlier, 'to estimate such rates. *But if the assumptions 

'A « 

are favorable to any sector it is probably the public sector. 

i ^ — \ 

* The evidence is thus rather strong that avejrage achievement 
growth- is considerably greater in the private sectors than ^ 
it is in the public sector [1:185]. V 

•' % 

A. SCHOOL POLICIES 
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„ We turn now to CHK's final phase, their "third method" for studying the dif- 
\ i 

ferential effects of^ public and private schools," which they call the "most 
valuable portion r of the analysis" [2:16, 23]. The full sweep of the claims has 
been cbnveyedJt>y Cplerean: * * ' 

V ✓ . • 

If Catholic or £?her private schools brirrg about higher 
achievement for ccjmparafcle students , ^and i_f they do 4o through 
* ^ those attributes measured^in the research [i.e., school policies] 
which distinguish Catholic ^and other private schools from public 
schools, then we should find achievement differences among 0 
schools within any sector, public or private.... ^Wjithin the 
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pufrlic ^sector, the performance of students similar to the 
/Average public fchool soptfomore, but with the levels ^frf homework an 
attendance . and disciplinary climate and student behavior 

1 ' " y 5 * 

attributable to school pofjcy in the Catholic or other 

• * .- 

private schools, the levels of achievement are approximately the 
same as those found in the Catholic and other private sectors, 

i 

The first i«plicatipfr tf of* these results is that they strongly 
Confirm the schbol-ef f ect remits found by the other two methods/ 
^For the selection Jj^nQthesis necessary to account for these differr- 
ences must ,be ( ej^<*ially tortured, operating not cmly J>etween 

. .sectors but? also to the same degree within sectors, *£n& operating 

, ,* 
to select students, on the -behavit>r variables indicated above, into 

r T ** * % 

* -schools With partlcul^^^rfiplinary climates,.. . -1 

A broa^gr imp li cat il\ holds well: . these attributes 

described a^bve are injtStt those which make a difference in 

achievement \ in -all AAericaryhigh schools no matter what sector 

-« they are iin. ' \Schdols which impose strong academic demands, 

schools whiph*ma£e*demands^on at tendance' and on behavior of 

^ students whfie tttly are in school are according to these 

pi results, scVools which bring about higher achievement [3:24-25]. 

' • "• • . - : ' ■ 

Our discuss ion >;i 11 first sketch' win aj: CHK did/ then, explain why their 
approach* wks wrong, and, finally,* explain why, if_ correct, their approach 
would lead to preposterous conclusl-ons. ' 

B. ' Procedures - 1 

The empirical core of this- phase of CHK's analysis consists of 

* . t , ' ' 

regressions of test-^core on backg^ptfnd and ptflicy variahl^s, run on the 



~ 41 



public sector only, separated by grade level: ^ 

(H) y - x'b + s'c + a • (j . } , 2 ) 



\ 



where y - fitted test score, x - background vector (17 variables), and s 
^school policy- vector (13 variables),. Auxiliary to* (11) are^the regressions ^of 
each of the school policy variables upon the background variables, which' may be 
represented in multivariate format as: * 

s 

* 

(.12) s' - P + h' • '(j-1,2) 

J' 

(13) I- -x' F 2j + lJ 2 2ijhij (J-1,2).- 

Here the F^ are matrices of slopes, while the are vectors of 
intercepfs. These equations are fitted separately at each grade level, to 
the public sector (12) and to the (combined) private sector (13). The 
auxiliary regressions are then evaluated at a common reference point, 
namely the, average public school sophomore's background, « to give 
background-controlled measures^of the policy vector as 

(14) qi t - # *h * t }+ \ (1-1,2,3; j-1,2) 
* i * 

(with F^j - P 2 j^ # Converting into increments by # 

> 

(15) .d tj - - qjj m (1.-2,3; j-1,2), 

we have the extra level of the policy variables 3et In each of^he private 
sectors, relative to the ptYbllc sector, after controllin^for measured 
background. Finally, CHK multiply these policy increments by the policy 

coefficients in (11) a'nd stim to^dbtain m 

* * /, / 

■ " 45 



42 • 

U6> 1 3 " ^1 V C ljk ; (i-2,3; 3-1,2),' 

w>iere k indexes the 1^ school policy variables. In words, the r* represent 
the predicted change in public sector test scores that would result if the 
public sector'policies were changed to the levels that prevail in private 
sector i *t grade level j* — all aftet controlling for background. 

^This is an elaborate procedure. CHK's previous use of regressions (see 
(3) - (4) above) would require them merely'to run equation (11) separately 
for each sector, evaluate those equations at a common reference point for x, 
and directly calculate the increments in y for the public sector after 
assigning private levels of s. CHK introduce (12) - (16) because they 
.recognize that the policy differences are themselves par attributable to 
background differences [1:209]. That the policies are not, aftet all, 

purely exogenous , is an Important point. 

* 

Indeed their route is even mo^e circuitous than so far described. They first 
, lead the reader through a maze of preliminary analyses [1:198-206] in which 
^test scores are regressed on the CHK 4 ^hort list" of five background variable^ 
and, in turn, several separate "school policy" variables. This maze implied n§rae * 
startling results. For example, one of the CHK school-policy variables, absen- 
teeism, shyfs an unusually large effect on test scores. Taking the regression 
results in Table 6.3.2 [1:2T)2] together with the variable coding [1:H3], 
we calculate that four days of acfditlorjal attendance per semester would raUe 
mathematics test scores by ,8!> points, Now', the predicted increase in mathemar- 
tics test scores between the sophomore -and senior years in* public schools 



etween the 
i^?Ta\] 
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is .88 (nee column TaT>le 3)'. Sp one has the implausible conclusion 

-that* four days of attendance per sera^ter are worth as much as the last two 
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years of high sch6ol i,n terms of test score gains. Implausible, that is, 
if one views the absenteeism variable as a measure of school policy. On 
the othenhand, if absenteeism is viewed as an 'indicator of student-specific 
traits — prior academic failing's, dislike of school, etc. — the finding i»^ 
entirely plausible. 

CHK also report the effect of school size on test scores [1:203-205]. The 
effect is positive, and public schools are larger than private schools. 

Since school size is a variable whic{i differs markedly across the sectors, 

• * 
it meets the criterion which CHK say they used in assembling their ' • 

policy vector, s [1:197].* But school size is also the only policy 
variable mentioned which is generally favorable to the public sector. And 
it is not included in their full policy vector, to which we now turn. 

There are thirteen variables in their full set of "school ^policies", 
all obtained from the student's questionnaire. It is instructive to note 
that this same set of variables, is variously ^referred to as "school charac- 
teristics and student behavioral -variables" [1 : 210] , "school functioning" ; 
variables [1:213; 2:21 ] ^ "school factors" [1:206, 207, 212], and "student 
behavior and schoe>«4j(mate^ S ^2: 19] . Grouped into 'five areas, the list is^ 
showj^J^nJJLx; Table 6.3.4; 2: Table 7] a >nd is discussed in some detail in 
[1:207]. The interested reader should consult the original survey questions 
from which those items were selected. Doing so will still leave one 
mystified as to why "disciplinary climate" is measured in part by students' 
ratings of their "teachers' interest in students." 

CHK's approach to jointly analyzing the effects of the thirteen school 

- • J 
policy ^riables upon test Scores is represented by our equations (11) - 

\ . * % V • k 

(16). In th^iT article, Table 7 [2:19} displays the <^^^ equation (15), 

47 • 
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while Table 8 [2:21] displays the r*^ of equation (16), In terms of com- 
ponents tn^lt up separately for each of the five policy "areas". In 
the Report, those displays appear as Table's 6.3.4 and 6.3,5 [l:2ia,. 213]. 
The underlying estimated regressions (11) - (13) are nowhere presented^ ft * 
Is evident, nonetheless, that some or all of the coefficients on the dis- 
ciplinary climate variables In the test score regression (11) were negative, 
at the sophomore level., That is to say, good disciplinary climate produced 
lower test scores. Noting this anomaly, CHK [1:216, 2:22] undertake a round- 
about subsidiary calculation [l:2l7 T 2^9], which, as far as we can determine/ 
amounts to re-estlmatlng equation fll) after excluding the student behavior 
variables. The results from this re-estlmation are hot shown but CHK refer 
to Table 6.3.6 [1:218] ln'clalmlng that most of the fe-estlmated coef- 
ficients on the disciplinary climate variables were positive. They then 
interpret this result as a demonstration that disciplinary climate operated 
through student behavior to produce, achievement [1:219; 2:23). It is not 
clear to us whether this subsidiary causal ordering -among their policy areas 
Is or is not valid. What is clear is that it will be Incorrect to credit 
both disciplinary climate and student behavior with positive effects upon 
achievement . 

The results for the final calculations (16) are shown in CHK f s Table 8 
[2:21]' and Tabled. 3.5 [1:215]. Their Table 8 is purported to show the 
achievement "differences within the public sector associated with the behavioT*!^ 
and school difference^ that remain between private and public schools when 

student backgrounds are controlled" [2:20]. The quoted passage may be 

r , 

unclear. Their reasoning is that if these Increments (shown in the 

M 0> * % 

accounted for" cows of their Table 8) are equal to the previously calcu- 
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lated backgrounjf-controlled increments (shown in the "overall- rows of their 
Table 8), then the test scores of public school students would equal those 
•of their hypothetical counterparts in the* private sector, if only'they were 
given the same level of the school policies. 

There are l\ pairs of numbers In CHK's Table 8 whidh 4 show the "total 

accounted for" and the ^overall" differences In test scores. Coleman's 

■ C ' 
claim Is that thr numbers in each pair are "approximately^ the same" [3:25]! 

As the reader can see, at most 6 of the 12 satisfy that description. 

Consider next the 7 pairs in which the "accounted for" number exceeds the 

"overall" number. CHK's interpretation should be that the public schools 

are more efficacious with these school policies than the private schools 

are. In fact their interpretation is that the analysis is "imperfect" and 

shows "puzzling differences" [1 : 214J^Nevertheless, the final judgment of 

r 

CHK Is that "the policy differences affect public achievement just as >tfhey 
do private school achievement" [2:23]* which leads Coleman to the sweeping 
claims we quoted jht the beginning of this section. 

CHK have not run regressions of test score on policy and background 
within the private sector. Nor have they offered any evidence that the 
coefficients on s are significantly different f rom zero in the public sec- 
tor. ' ' y 

C. Critique 4 s 

We advise readers not to devote much energy puzzling over the issuea^of 
ambiguity of definition, sector effica^r, or statistical significance /' 
regarding the CHK policy variables. The .variables do not define, and /only 
remotely* reflect, school policies. The variables are the student • s d/scrip- 
tlon of his or her personal behavior and perceptions of others* ^behavior. 



43 




46 

In coWfast to these itetns are some HS6B questions asked of school officials 
that vfrre about "school policies. But these responses make no appearance* in 

, the CHK analysis . ^ 

- V 

The School policy" variables used by CHK* have all the appearance of 

c 

teing £a) primarily, reflections of studenl background characterist ics* jiot . 
otherwise controlled fqr; and (b) $econdarily, endogenous outcomes 
reflecting school achievement; and (c) least of all, exogenous school policies. 
In this light, we see that CHK are attributing to the "public schools nega- 
tive veffects that reflect sources (a) and (b). 
t Consider the variables defined by whether or not the student has taken an ad- 
vanced mathematics course or an honors course in mathematics or in English. 

\re % these not indicators of academic abilities and motivations 9 9 Are these 

4 

not also measures of scholastic achievement, just as the test scores them- 
>* 

selves are?, "Having taken an advanced mathematics course" is a variable 

'•V 

whose role i* a regression equation explaining mathe/uatics^test scored is 
similar to that of a* variable defined as "having achieved a high grade in a 
previous mathematics course." Does ^fhe latter represent a school polity? 
Yes, to some extent. But is it sensible t?o estimate the effect of 
school policies with such a variable? (Vote that the variable for having 
taken an advanced mathematics course Is not defined by whether or not t^e, 
school of f ers the course.) 

"Doing homework," "attending school , "^and "cutting class" are similarly 
aspects of a student's behavior that, At face value, are predominantly reflet- 

L j 

tions of the student's motivations and academic orientation. Are we to 
believe that the role the school has . played^ in shaping the student's motiva- 
tion and orientation justifies calling these variables "school policies" as 

* „ 

t 
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soorf as their dependence on 17 measured background variables has been 
extracted? No doubt some of the variation in the Ch% school policy 
variablea^is explained by the 17 background variables. CHK do not tell us 
how much, but \t does not reaUy aatter , because the 13 so-called policy 
variables continue to reflect other background characteristics of the students. 
Recall that analysis of covariance # on measured background is in their words a 
"particularly defective" method for" controlling fof such dif f icult-to-measure 
concepts as student motivation and ability. 

Indeed, CHK themselves raise the issue that we have been 
discussing: 

One might argue that . .. the kind of students who tend to be 

/ 

lower achievers are those who are absent or cut classes, and 
it is not the absences themselves that reduce achievement. 
This may be so, an3 the issue certainly merit's further 
attention [1:200]. %t 

* 

But they dismiss this argument in the very next sentence on the grounds that 
the regression coefficients of y on s are similar "in the different sectors, 
where policies lead to very'dif ferent levels of absenteeism." The force of * 
this logic escapes us entirely. 

Two additional wrinkles *in the CHK patchwork of school-policy variables 
need to be discussed. First, CHK construct school means for "cutting class"* 
and for "absenteeism" that omit the student's own responses, and assign these 
school means to the student [1:215-216]. Evidently, all student:- in a 
school are assigned approximately the same 'value for the cl^ss-cutt ing 
and the absenteeism variables. CHJC would have us believe that this purges the 
variable^ of their background component. But a *simple example should 

: r - 
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m ., '* 

dispose of this spurious claim. , « 

Assume earnings for the ith worker, a t - the worker' s. age, and a 

regression produces dy 1 /da 1 > 0 as evidence that * i haa^a positive effect on 
y 1 - Suppose another hypothesis is that a firm's "age pol??^ has an effect ' 



on earnings. To test-tphis, y is regressed on a, where a is the mean age of 

the ith worker's co-workers. It is found that dj./da > 4 Q, Is this evi- 

/' " , 

dence for the hypothesis that a firm's "age policy" has a positive effect on 

the earnings of a given worker? Surely a counterinterpretation is that a 
is a proxy for a^. In fact if a 1 were a test score, it might be so fallible 
a measure that a could be a ^better estimate of the true score, A natural 
way to proceed is to_£egress y on a t and a, which will indicate whether a 
has*an effect after controlling for a^ Evidently, CHK would^ind it suf- 
ficient to regress on a and some representing the worker's 
"background," and then assert that a shows the effect of the firms's "age 
policy," , # 

A Becqnd vfr inkle ~int roduced, by CHK in j:heir quest for a "school policy" 
interpretat Jlofc- is to rely on the variables that define the student's percep- 
tion of other students' behavior. They claim thajT # 

t 

disciplinary climate ... and student behavior in the 4$ 
school ... characterize the school as a unit, rather ^than 
the student. They are least susceptible to the alternative 
selection hypothesis, which for them must become especially 
tortured [2:18]. 

By thifr logic, any school-level variable that is constructed by averaging^ 
students' characteristics — e.£., the students' mean score, on a mathernatics 
test takerTln eighth grade — is on the same footftag as a school variable that 
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is an actual policy — e.g., a policy 'of not giving grades. ' CHK could' say, 
after all, that both variables -characterize the school as a unit " and 
going tfn^e offensive, assert that it would be "especially tortured" to 
contend that eighth grade test scores reflect initial differences 
among the students entering the high schools. * 

With their disciplinary climate variables CHK have shifted attention 
from school policy variables towards school-level variables of whatever 
nature. Their use of a student-behavior variable, which includes such beha- 
vior as "attacking teachers, " presents a dlf fere^l^j£es^t^^ namely, Is a ' 
student's test score affected by the background characteristics of his'or 
her peers? This is indeed an important Issue for many parents*, but it does 
not address the question, What is the effect on a student's test score of 
school ^ticies? In passing, we might note that an improvement to'the CHK*~ 4 " 
specification to measure the effect of the Deer group's disciplinary beba- 
vior is a regression that also includes as a variable tjie student 1 s^own 
prior disciplinary conduct. The peer-group effect would generally be biased 
if the Individual effect were not controlled, just as, in the example above, 



) 



•r f 



.dy^/da was biased in the absence of a control for a^ 



D. Assessments . - ✓ 
— * 

^n assembling- their Jlst, of 13 schooTpolicy variables, CHK selected some 
items from the questionnaire, but not* others. In particular, in defining the 
"coursewojk" variable, they chose (a) the algebra, geometry, and trigonometry 

it not the calculus, physics, or chemistry courses; and (b) "advanced 



or honOTs or 



or honffs program" in English or mathematics, but not "remedial programs" 
i^gngtish or mathematics. * ,< 

We do not know wh*J!HK made these choices. Had they used the items they 
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passed over,^e suspect that they would have reached the following conclusions: 
(1) students who have taken, or are taking, calculus score higher on the 
mathematics test, even after controlling for background; (2) students who 
have taken, or are taking, remedial mathematics courses score lower on the 
mathematics test, even after controlling for background. Consequently, it~ 
CHK were to adhere to their position that "coursework" is a school policy, 
they woul<J conclude that removing* students from remedial math courses and/or 
putting fchera into^alculus purses would raise mathematics test scores by 
the amounts shown by the coefficients in their regressions. 

, Had CHK presented such findings on the calculus and remedial variables, 
the findings, at least, would have been met with derision. Should the reac- #► 
t-ion be any different because they are using "advanced math" and "honors" 
variables, .instead? , 

A final note: Recall Coleman's stance that "academic track" is a policy 
variable, rather than a proxy for the background abilities and motivations of the 

' iff * . 

students. If so, shouldn't It have been included In the policy vector? Had N 
t<hat t>e«n done, another powerful policy would have been revealed: just 
shift students from vocational and general tracks -to academic tracks and 
secure the gains measured by the coefficients on tVack! The fallacy L 
li\volved would be precisely the same as> In the CHK interpretation of their school 
policy variables. 

5. CONCLUSION , 

In our assessment of trffe CHK analysis of cognitive outcomes we have 
found no basis for accepting their conclusions and no merit in their analy- 
sis. Their principal conclusions about the cognltivr^gains Attributable *o 

A ' - t 

the private jfcctor and to certain school policies are not warranted by 



^ their data. Their research" methods, or r alternatively, their execution of 
^ :<f *tt;he methods, are replete with flaws* The presentation of {tieir analyses is 
confusing, incorapflfete, and bjased. . Their mistakes and their style are one- 
sid-ed, --^pro-private. It is as \f they decided <to write a brief for the 
proposition that socie^giift to -the subsidization of private schools and 

away trom the subsidization of public schools. These faults are also evi- 

< 

* » • 

dent In other aspects of their Report: (a) ,the\relation between private 
schoqling and segregation (see Taeuber^fmT James [15]); and (b) the relation 
^^l" * nC( ' me and tultlon tax-credits, 05 the one hand, and attendance/at, 
or choice of, private qt public schools, on the otner (see Catterall and Levin 
j [7f^and Gdldberger [16]). r \ " 

^ . CHK conVlud? on the Iwsis of four, phases of analyses that the privafe 
sector has beneficial effects, on cognitive achievement. We summarize our 



assessment of these analyses as Allows*:' 



s 



1. Their first analysis teed multiple regression with 17 background 

variables,. Our 'discus* icm, along with tha,t of prev^us commentators (see 

** * * * ' 

Educational Research Service [17] and Murnane [18]), noted the presumptive bias 

• 0 

* against puMic shools in this approach. When academic track is .used as a 

- reasonab"alternative control for this selection bias, the outcomes in favor of 
^the ^pri^te. sector virtually disappear. 

2. A second analysis purported to show that^ Ca tholic schools- are * 
^m6re eg<*li Ur^T^c au s e they produce more similar te'st scores^among stu- * 

'dents wfrth^fverse s^ioeconqmic backgrounds. This approach was twice 

first, by fche exclusion of 'relevant hackgrftund variables (12 of the 
%eci>nd, we surmise, by the truncation of student cognitive abilities in 
a narrower range in the Catholic sector than in the putflic sector. 

C ' * ' J 
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3. The trhird analysis was the study of sophomore- to-senior 
change in test scores. The CHK ^pro-private conclusion here was duetto the 

absence of control over background. Our measured sophoraore-to-senior change, 

« * 

using backgroul^d controls, showed that the advantage to the private sector 

disappears (i€f a conventional growth rate formula is used) or is* sharply 
attenuated (if CHK's ceiling formula is used). The background-control 

approach is itself not adequate, however, so its results are. only indicative 

• /- 
of the direction of bias in the CHK approach. 

A. CHKfs final analysis, in which thJ^ attribute the test A score gains in 

• ^ ' \ > 

the private sector to seigcte^ school policies, strikes us.as patently falla- 

f *» ^ 

clouds. School policies get credit for outcomes that are due to studenp 

backgrounds. In this section CHK push their" pro-private methods and 

V 

interpretations to fanciful extremes. For CHK,' taking an advanced mathematic 
Course and, for Coleman, taking the academic ' tAck, ate "school policies." 

And to measure the effects of such "policies" dn test scores, they merely 

\ 

read the coefficients of their "policy -variables from regression^ that also 
include the background variables. The pitfalls of selection are apparent. 
What 19 true of the variables "advartc^l' mathematics course" and "academic 
track" is true df "private sector" ; name ly , they all contain an obvious bias 
due to omitted srtudent ability and background characteristics. 

>• •' ; v * . • 
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